Asymmetry compensation for private line emulation

ABSTRACT

In one example, an indication of a time during which a network communication obtained from a first network node was processed by the first network node, and an indication of a propagation delay from a second network node to the first network node, are obtained. A time during which the network communication was processed by the second network node is determined. A propagation delay from the first network node to the second network node is calculated based on the time during which the network communication was processed by the first network node and the time during which the network communication was processed by the second network node. A difference between the propagation delay from the first network node to the second network node, and the propagation delay from the second network node to the first network node, is determined and compensated is made for that difference.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/955,549, filed Dec. 31, 2019, the entirety of which is incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure relates to telecommunications.

BACKGROUND

Private Line Emulation (PLE) is a framework that allows for bittransparent transport of high speed signals over a Multi-Protocol LabelSwitching (MPLS) infrastructure. PLE is similar to circuit emulationconcepts such as Structure-Agnostic Time-Division Multiplexing (TDM)over Packet (SAToP), Circuit Emulation Services over Packet (CESoP) andCircuit Emulation over Packet (CEP) that have been defined in the pastfor carrying Synchronous Optical Networking (SONET)/Synchronous DigitalHierarchy (SDH)/Plesiochronous Digital Hierarchy (PDH) signals. PLEleverages Virtual Private Wire Service (VPWS) packet encapsulation anddecapsulation and Differential Clock Recovery (DCR) to maintain theclock of the transported signal (frequency). As such, PLE is analternative to deploying an Optical Transport Networking (OTN) networkfor delivering similar services such as transparent 10 Gbps connections.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a Private Line Emulation (PLE) network configured forasymmetry compensation, according to an example embodiment.

FIGS. 2A and 2B illustrate representations of forward and reversepropagation delays in a PLE network, according to an example embodiment.

FIG. 3 illustrates a functional diagram depicting PLE network asymmetrycompensation, according to an example embodiment.

FIG. 4 illustrates a network communications diagram for asymmetrycompensation in a PLE network, according to an example embodiment.

FIG. 5 illustrates a block diagram of a PLE network node configured forasymmetry compensation, according to an example embodiment.

FIG. 6 illustrates a flowchart of a method for asymmetry compensation ina PLE network, according to an example embodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

In one example embodiment, an indication of a time during which anetwork communication obtained from a first network node was processedby the first network node, and an indication of a propagation delay froma second network node to the first network node, are obtained. A timeduring which the network communication was processed by the secondnetwork node is determined. A propagation delay from the first networknode to the second network node is calculated based on the time duringwhich the network communication was processed by the first network nodeand the time during which the network communication was processed by thesecond network node. A difference between the propagation delay from thefirst network node to the second network node, and the propagation delayfrom the second network node to the first network node, is determined.Compensation is made for the difference between the propagation delayfrom the first network node to the second network node, and thepropagation delay from the second network node to the first networknode.

Example Embodiments

FIG. 1 illustrates an example Private Line Emulation (PLE) network 100configured for asymmetry compensation. PLE network 100 includes MPLScore 105, network nodes (e.g., Provider Edge devices (PEs)) 110(1) and110(2), Attachment Circuits (ACs) 115(1) and 115(2), and Customer Edgedevices (CEs) 120(1) and 120(2). MPLS core 105 further includes networknodes 125(1)-125(3), which form an MPLS Label-Switched Path (MPLS-LSP)130 between PEs 110(1) and 110(2). Pseudowire (PW) 135 may emulate aphysical wire through a point-to-point connection between PEs 110(1) and110(2). Thus, CEs 120(1) and 120(2) may communicate with one another bysending one or more network communications (e.g., network frames,network packets 140(1)-140(7), etc.) over PW 135. PW 135 may be mappedto MPLS-LSP 130 in order to carry out the network communication for PW135 across the MPLS core 105.

In one example, MPLS core 105 runs an Interior Gateway Protocol (IGP)such as Open Shortest Path First (OSPF) or IntermediateSystem-to-Intermediate System (IS-IS). MPLS core 105 may also run atraffic engineering protocol such as Resource Reservation Protocol forTraffic Engineering (RSVP-TE) or Segment Routing for Traffic Engineering(SR-TE). PW 135 may be established via Targeted Label DistributionProtocol (T-LDP) for Virtual Private Wire Service (VPWS), and ACs 115(1)and 115(2) may utilize Synchronous Optical Network (SONET), Ethernet, orOptical Transport Networking (OTN) technology. However, any suitableparadigm/protocol may be utilized in conjunction with the techniquespresented herein. For example, any suitable tunneling technology may beused instead of MPLS, such as Layer 2 Tunneling Protocol (L2TP), UserDatagram Protocol (UDP), Internet Protocol (IP), etc. As an alternativeto MPLS core 105, the core may be based on IP or Segment Routing over IPversion 6 (SRv6).

Initially, PEs 110(1) and 110(2) may be configured to send/receiveoptical signals to/from each other based on synchronized systemfrequencies (e.g., synchronized local clocks 145(1) and 145(2)). Localclocks 145(1) and 145(2) may be synchronized via any suitable mechanism,such as external timing input (e.g., Building Integrated Timing Supplies(BITS), 10 MHz, etc.), Synchronous Ethernet (SyncE), Precision TimeProtocol (PTP), etc., based on reference clock 150. For example, PE110(1) may obtain a network communication (e.g., a network frame) fromCE 120(1) via port 155(1), add (e.g., via encapsulation) to the networkcommunication an RTP counter based on local clock 145(1), and send thenetwork communication to PE 110(2) over PW 135 as a network packet. PE110(2) may determine the RTP counter from the network communication(e.g., via decapsulation), and use the RTP counter to performDifferential Clock Recovery (DCR) as shown at reference numeral 158before sending the network communication (e.g., as a network frame) toCE 120(2) via port 155(2).

The network packet referenced in the aforementioned example may benetwork packet 140(7). As shown, network packet 140(7) is astructure-agnostic emulation packet that includes Packet SwitchedNetwork (PSN) header 160, Real-time Transport Protocol (RTP) header 165,control word 170, and payload 175. PSN header 160 may direct networkpacket 140(7) to PE 110(2) via PW 135. RTP header 165 may be used fortransfer of timing information. Control word 170 may identify a sequencenumber used to provide a common PW sequencing function as well local andremote failure indication between PE 110(1) and PE 110(2). RTP header165 or control word 170 may include the timestamp referenced in theaforementioned example. Payload 175 may include a fixed number of octetsreceived from AC 115(1). It will be appreciated that network packets140(1)-140(6) may have similar structures, and that PE 110(2) may sendone or more similarly structured network packets to PE 110(1). However,any suitable network packet structure may be employed in accordance withtechniques presented herein.

Synchronization techniques such as external timing input, SyncE, PTP,etc., may help synchronize local clocks 145(1) and 145(2), but theycannot ensure that the propagation delay from PE 110(1) to PE 110(2)(also referred to herein as “the forward propagation delay”) isidentical to the propagation delay from PE 110(2) to PE 110(1) (alsoreferred to herein as “the reverse propagation delay”). In other words,such synchronization techniques cannot ensure symmetric propagationdelay in PLE network 100. As used herein, the term “propagation delay”may refer to an amount of time for a network packet to travel from onenetwork node from another. Asymmetric propagation delay can arise, forexample, when an optical fiber used to send network communications inone direction (e.g., from PE 110(1) to 110(2)) has a greater length thanan optical fiber used to send network communications in anotherdirection (e.g., from PE 110(2) to PE 110(1)). Other causes ofasymmetric propagation delay may include differences in forwarding delaythrough devices or the use of Digital Signal Processing (DSP) (e.g.,coherent DSP), transmission technologies for links in MPLS core 105, thetemperature of the optical fiber(s), etc.

Propagation delay may be measured based on any suitable start time andend time. For example, the propagation delay may be measured starting ator near ingress of a network packet to PE 110(1)/110(2) and ending at ornear egress of a network packet to PE 110(2)/110(1). However, there areother options for measuring propagation delay (e.g., starting at or nearingress of a network packet to PE 110(1)/110(2) and ending at or nearingress of a network packet to PE 110(2)/110(1), starting at or nearegress of a network packet to PE 110(1)/110(2) and ending at or nearegress of a network packet to PE 110(2)/110(1), etc.). More generally,the start time of propagation delay may be any suitable time duringwhich a network packet is being processed by PE 110(1)/110(2), and theend time of propagation delay may be any suitable time during which anetwork packet is being processed by PE 110(2)/110(1).

Asymmetric propagation delay is a growing problem with the advent of 5Gnetworking. In particular, the industry is struggling to meet stringent5G time synchronization requirements demanded by the transportinfrastructure of accuracies on the order of 100s (hundreds) ofnanoseconds. Optical Transport Networking (OTN) switching, for example,is commonly used by carriers to provide transparent 10 Gbps connectionsto mobile operators, but this introduces significant delay asymmetries.These delay asymmetries can prevent the mobile operator from achievingadequate time synchronization accuracy because PTP recovery (forexample) is based on the fundamental assumption that the connectionsbetween nodes have perfect delay symmetry.

Accordingly, asymmetric delay compensation logic 180(1) and 180(2) areprovided on PEs 110(1) and 110(2), respectively, to address asymmetryproblems in connections delivered over legacy networks (e.g., legacy OTNnetworks). In one example, asymmetry correction is made in the egressbuffer of a PE by measuring and encoding time and delay informationOut-Of-Band (00B) or in a Private Line Emulation (PLE) header. Theegress buffer may buffer network communications for any suitable amountof time (e.g., on the order of nanoseconds or microseconds). In oneexample, egress buffer may be used for asymmetry compensation as well asde jitter to address Packet Delay Variation (PDV) (e.g., between 200 and400 microseconds). Asymmetric delay compensation logic 180(1) and 180(2)thus enables emulation of a virtual fiber or physical transparentconnection that has symmetric propagation delay, even where theunderlying network is not symmetric. Asymmetric delay compensation logic180(1) and 180(2) may utilize reference time of day 185 to align localtimes of day 190(1) and 190(2) (e.g., using PTP or external time of dayinput). Unlike local clocks 145(1) and 145(2) and reference clock 150,which are used to track frequencies on behalf of PEs 110(1) and 110(2)in accordance with conventional approaches, reference time of day 185and local times of day 190(1) and 190(2) enable PEs 110(1) and 110(2) tosignal timestamps to each other in accordance with techniques describedherein.

With continuing reference to FIG. 1, FIGS. 2A and 2B illustratediagrammatic representations 200A and 200B of forward and reversepropagation delays Δt_(F) and Δt_(R) in PLE network 100. As shown inFIG. 2A, PE 110(1) obtains, via port 155(1), network frames205(1)-205(3), and sends (e.g., via PW 135) corresponding networkpackets 210(1)-210(3) to PE 110(2), which outputs network frames205(1)-205(3) via port 155(2). Similarly, PE 110(2) obtains, via port155(2), network frames 205(4)-205(6), and sends (e.g., via PW 135)corresponding network packets 210(4)-210(6) to PE 110(2), which outputsnetwork frames 205(4)-205(6) via port 155(1). Network frames205(1)-205(3) may have a frequency f₁ on port 155(1) and a frequency off₂ on port 155(2). Network frames 205(4)-205(6) may have a frequency f₃on port 155(2) and a frequency of f₄ on port 155(1).

As shown in FIG. 2B, forward propagation delay Δt_(F) is measuredbeginning when network frame 205(1) is first received at port 155(1),and ending when network frame 205(1) is first transmitted from port155(2). Similarly, reverse propagation delay Δt_(R) is measuredbeginning when network frame 205(4) is first transmitted from port155(2), and ending when network frame 205(4) is first received at port155(1). DCR 158 may be used to ensure that the frequency f₁ of networkframes 205(1)-205(3) on port 155(1) is identical (or substantiallysimilar) to the frequency f₂ of network frames 205(1)-205(3) on port155(2), and to ensure that the frequency f₃ of network frames205(4)-205(6) on port 155(2) is substantially similar to the frequencyf₄ of network frames 205(4)-205(6) on port 155(1). However, as noted,DCR 158 (and other existing synchronization techniques) do notcompensate for asymmetry between forward and reverse propagation delaysΔt_(F) and Δt_(R).

With continuing reference to FIG. 1, FIG. 3 illustrates a functionaldiagram 300 depicting PLE network asymmetry compensation. Briefly, PEs110(1) and 110(2) both perform two functions, delay calculation andasymmetry compensation, with PE 110(1) responsible for calculating thepropagation delay from PE 110(2) to PE 110(1) (i.e., reverse propagationdelay Δt_(R)), and PE 110(2) responsible for calculating the propagationdelay from PE 110(1) to PE 110(2) (i.e., forward propagation delayΔt_(F)).

In one example, PE 110(2) may obtain network communication 305 (e.g., anetwork frame) via port 155(2) and insert an indication of a first timet₁ during which network communication 305 was processed by PE 110(2)based on local time of day 190(2). For example, PE 110(1) may obtainnetwork communication 305 from CE 120(1), packetize data from networkcommunication 305 to a payload, and encapsulate the payload with acontrol word, RTP header, and PSN header. The first time t₁ may be atime at or near ingress of network communication 305 to port 155(2). PE110(2) may provide network communication 305, including the indicationof the first time t₁, to PE 110(1). The indication of the first time t₁may be inserted as a timestamp comprising an RTP timestamp, a controlword, or a dedicated field of network communication 305.

This indication may enable PE 110(1) to calculate Δt_(R). For example,PE 110(1) may determine a second time t₂ during which networkcommunication 305 was processed by PE 110(1) based on local time of day190(1). In one example, the second time t₂ may be a time at or nearingress of network communication 305 to PE 110(1). PE 110(1) may furtherdecapsulate network communication 305, determine t₁, and perform delaycalculation to calculate the difference between t₂ and t₁ (i.e.,Δt_(R)). Thus, PE 110(1) is now aware of Δt_(R).

PE 110(1) may obtain network communication 315 (e.g., a network frame)via port 155(1) and insert (e.g., via encapsulation) an indication of athird time t₃ during which network communication 315 was processed by PE110(1) based on local time of day 190(1), and an indication of Δt_(R).For example, the third time t₃ may be a time at or near ingress ofnetwork communication 315 to port 155(1). PE 110(1) may provide networkcommunication 315, including the indication of the third time t₃ and theindication of Δt_(R), to PE 110(2). The indication of the third time t₃may be inserted as a timestamp comprising an RTP timestamp, a controlword, or a dedicated field of network communication 315. The indicationof Δt_(R) may be inserted in a PLE header of network communication 315.

The indication of the third time t₃ may enable PE 110(2) to calculateΔt_(F). For example, PE 110(1) may determine a fourth time t₄ duringwhich network communication 315 was processed by PE 110(2) based onlocal time of day 190(2). In one example, the fourth time t₄ may be atime at or near ingress of network communication 315 to PE 110(2). PE110(2) may further decapsulate network communication 315, determine t₃,and perform a delay calculation to calculate the difference between t₄and t₃ (i.e., Δt_(F)). Thus, PE 110(1) is now aware of both Δt_(R) andΔt_(F).

PE 110(2) may obtain network communication 320 (e.g., a network frame)via port 155(2) and insert (e.g., via encapsulation) an indication ofΔt_(F). The indication of Δt_(F) may be inserted in a PLE header ofnetwork communication 320. PE 110(2) may provide network communication320, including the indication of Δt_(F), to PE 110(1). PE 110(1) maydecapsulate network communication 320 and determine Δt_(F). Thus, PE110(1) is now also aware of both Δt_(R) and Δt_(F), and PEs 110(1) and110(2) may determine a difference between Δt_(R) and Δt_(F) and performasymmetry compensation to compensate for the difference therebetween.For example, if Δt_(F) is less than Δt_(R), PE 110(2) may delay networkcommunication 315 in egress buffer 310(2), and if Δt_(F) is greater thanΔt_(R), PE 110(1) may delay network communication 320 in egress buffer310(1).

More specifically, if Δt_(F) is less than Δt_(R), PE 110(2) may delaynetwork communication 315 in egress buffer 310(2) for an amount of timesubstantially equal to the difference between Δt_(F) and Δt_(R), and PE110(1) may refrain from delaying network communication 320 in egressbuffer 310(1). Conversely, if Δt_(F) is greater than Δt_(R), PE 110(1)may delay network communication 320 in egress buffer 310(1) for anamount of time substantially equal to the difference between Δt_(R) andΔt_(F), and PE 110(2) may refrain from delaying network communication315 in egress buffer 310(2). PEs 110(1) and 110(2) thereby compensatefor any asymmetry between Δt_(F) and Δt_(R) by ensuring that propagationdelays in the forward and reverse directions are substantially the same.

Because asymmetry between Δt_(F) and Δt_(R) may be dynamic (e.g., maychange relatively slowly or quickly over time), in one example, Δt_(R)and/or Δt_(F) may be updated continuously (e.g., with every networkcommunication sent between PEs 110(1) and 110(2)), periodically,randomly/pseudorandomly, or in response to certain triggers (e.g., achange in network load, a change in network design, a change intemperature, etc.). For example, PE 110(2) may also insert (e.g., viaencapsulation) in network communication 320 a fifth time t₁′ duringwhich network communication 320 was processed by PE 110(2) based onlocal time of day 190(2). The fifth time t₁′ may be a time at or nearingress of network communication 320 to port 155(2). This indication mayenable PE 110(1) to update Δt_(R). The indication of the fifth time t₁′may be inserted as a timestamp comprising an RTP timestamp, a controlword, or a dedicated field of network communication 320.

Further still, PE 110(1) may determine a sixth time t₂′ during whichnetwork communication 320 was processed by PE 110(1) based on local timeof day 190(1). In one example, the sixth time t₂′ may be a time at ornear ingress of network communication 320 to PE 110(1). PE 110(1) maydecapsulate network communication 320, determine t₁′, and perform delaycalculation to calculate the difference between t₁ and t₁′ (i.e.,updating Δt_(R)). PE 110(1) may provide an indication of the updatedΔt_(R) to PE 110(2) in further network communications.

In the specific examples described herein, indications of times andΔt_(R) and Δt_(F) are transmitted in-band (e.g., in networkcommunications 305, 315, and 320). However, it will be appreciated thatsuch indications may also or alternatively be transmitted out-of-band.For example, the indications of times and Δt_(R) and Δt_(F) may betransmitted in separate/independent/dedicated network communications(e.g., in a packet that does not also transport PLE data).

Here, the carrier operating MPLS core 105 is using asymmetrycompensation to enforce identical latency in both directions of theconnection established via PW 135 between PE 110(1) and PE 110(2). Thisensures that the assumption of the mobile operator (e.g., that theforward and reverse propagation delays are symmetric) holds and that notime recovery errors are introduced within the mobile operatorsinfrastructure that is built using the connection delivered via thecarrier. Furthermore, the carrier and mobile operator need not interactwith regard to time synchronization. Actions taken by the mobileoperator may be transparent to the carrier, and hence the carrier mayserve multiple mobile operators. The carrier may ensure a symmetricconnection so that the time synchronization of each mobile operator isprecise and independent from other mobile operators. Without PLE and theasymmetry compensation techniques described herein, the mobile operatorswould need to rely on the carrier providing timing as a service and beno longer independent from the carrier and other mobile operators. Thecarrier would also need to deal with interactions between multiplemobile operators, created an overall a dissatisfactory situation for allinvolved parties.

With continued reference to FIGS. 1 and 3, FIG. 4 illustrates an examplenetwork communications diagram 400 for asymmetry compensation in PLEnetwork 100. At 410, PE 110(2) provides, to PE 110(1), networkcommunication 305 (including an indication of the first time t₁). At420, PE 110(1) provides, to PE 110(2), network communication 315(including indications of the third time t₃ and Δt_(R)). At 430, PE110(2) provides, to PE 110(1), network communication 320 (includingindications of the fifth time t₁′ and Δt_(F)). This is consistent withthe example of FIG. 3, although operations may be performed in anysuitable order to compensate for asymmetry between Δt_(R) and Δt_(F).

For example, PE 110(1) may provide an indication of a time at which thenetwork communication was processed by PE 110(1), after which PE 110(2)may provide an indication of Δt_(F) to PE 110(1), after which PE 110(1)may provide an indication of Δt_(R) to PE 110(2). Alternatively, PEs110(1) and 110(2) may provide, to each other, indications of times atwhich respective network communications were processed thereat, afterwhich PEs 110(1) and 110(2) may provide, to each other, indications ofΔt_(R) and Δt_(F). It will be appreciated that indications of the timesand propagation delays may be sent together (e.g., in networkcommunications 315 and 320) or separately (e.g., in-band and/orout-of-band). Still other embodiments may be envisioned.

FIG. 5 is a block diagram of an example optical node 500 (e.g., PEs110(1) or 110(2)) that is configured to implement techniques presentedherein. Optical node 500 includes a network interface in the form ofoptical receiver 510 and optical transmitter 520 (including optical andelectrical modules) that enable communications over an optical networkvia one or more optical fibers 530(1) and 530(2), Application SpecificIntegrated Circuit (ASIC) 540 (or multiple such ASICs), which performsnetwork processing functions, one or more processors 550, and memory560. Processor(s) 550 may include any suitable processing entity, suchas microprocessors, microcontrollers, Field-Programmable Gate Arrays(FPGAs), Application-Specific Standard Parts (ASSPs), etc. Otherembodiments may be envisioned.

Memory 560 may include read only memory (ROM), Random Access Memory(RAM), magnetic disk storage media devices, optical storage mediadevices, Flash memory devices, electrical, optical, or otherphysical/tangible memory storage devices. Thus, in general, memory 560may comprise one or more tangible (non-transitory) computer readablestorage media (e.g., a memory device) encoded with software comprisingcomputer executable instructions and when the software is executed(e.g., by the one or more processors) it is operable to performoperations described herein. In particular, memory 560 includesasymmetric delay compensation logic 180(i) (e.g., asymmetric delaycompensation logic 180(1) or 180(2)) that, when executed, enablesoptical node 500 to perform the techniques described herein.

FIG. 6 is a flowchart of an example method 600 for compensating forasymmetric propagation delays. Method 600 may be performed, for example,by PE 110(1) or PE 110(2). At 610, an indication of a time during whicha network communication obtained from a first network node was processedby the first network node, and an indication of a propagation delay froma second network node to the first network node, are obtained. At 620, atime during which the network communication was processed by the secondnetwork node is determined. At 630, a propagation delay from the firstnetwork node to the second network node is calculated based on the timeduring which the network communication was processed by the firstnetwork node and the time during which the network communication wasprocessed by the second network node. At 640, a difference between thepropagation delay from the first network node to the second networknode, and the propagation delay from the second network node to thefirst network node, is determined. At 650, compensation is made for thedifference between the propagation delay from the first network node tothe second network node, and the propagation delay from the secondnetwork node to the first network node.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment. However, itshould be appreciated that any particular program nomenclature herein isused merely for convenience, and thus the embodiments should not belimited to use solely in any specific application identified and/orimplied by such nomenclature.

Data relating to operations described herein may be stored within anyconventional or other data structures (e.g., files, arrays, lists,stacks, queues, records, etc.) and may be stored in any desired storageunit (e.g., database, data or other repositories, queue, etc.). The datatransmitted between entities may include any desired format andarrangement, and may include any quantity of any types of fields of anysize to store the data. The definition and data model for any datasetsmay indicate the overall structure in any desired fashion (e.g.,computer-related languages, graphical representation, listing, etc.).

The present embodiments may employ any number of any type of userinterface (e.g., Graphical User Interface (GUI), command-line, prompt,etc.) for obtaining or providing information, where the interface mayinclude any information arranged in any fashion. The interface mayinclude any number of any types of input or actuation mechanisms (e.g.,buttons, icons, fields, boxes, links, etc.) disposed at any locations toenter/display information and initiate desired actions via any suitableinput devices (e.g., mouse, keyboard, etc.). The interface screens mayinclude any suitable actuators (e.g., links, tabs, etc.) to navigatebetween the screens in any fashion.

The environment of the present embodiments may include any number ofcomputer or other processing systems (e.g., client or end-user systems,server systems, etc.) and databases or other repositories arranged inany desired fashion, where the present embodiments may be applied to anydesired type of computing environment (e.g., cloud computing,client-server, network computing, mainframe, stand-alone systems, etc.).The computer or other processing systems employed by the presentembodiments may be implemented by any number of any personal or othertype of computer or processing system (e.g., desktop, laptop, PersonalDigital Assistant (PDA), mobile devices, etc.), and may include anycommercially available operating system and any combination ofcommercially available and custom software (e.g., machine learningsoftware, etc.). These systems may include any types of monitors andinput devices (e.g., keyboard, mouse, voice recognition, etc.) to enterand/or view information.

It is to be understood that the software of the present embodiments maybe implemented in any desired computer language and could be developedby one of ordinary skill in the computer arts based on the functionaldescriptions contained in the specification and flow charts illustratedin the drawings. Further, any references herein of software performingvarious functions generally refer to computer systems or processorsperforming those functions under software control. The computer systemsof the present embodiments may alternatively be implemented by any typeof hardware and/or other processing circuitry.

The various functions of the computer or other processing systems may bedistributed in any manner among any number of software and/or hardwaremodules or units, processing or computer systems and/or circuitry, wherethe computer or processing systems may be disposed locally or remotelyof each other and communicate via any suitable communications medium(e.g., Local Area Network (LAN), Wide Area Network (WAN), Intranet,Internet, hardwire, modem connection, wireless, etc.). For example, thefunctions of the present embodiments may be distributed in any manneramong the various end-user/client and server systems, and/or any otherintermediary processing devices. The software and/or algorithmsdescribed above and illustrated in the flow charts may be modified inany manner that accomplishes the functions described herein. Inaddition, the functions in the flow charts or description may beperformed in any order that accomplishes a desired operation.

The software of the present embodiments may be available on anon-transitory computer useable medium (e.g., magnetic or opticalmediums, magneto-optic mediums, floppy diskettes, Compact Disc ROM(CD-ROM), Digital Versatile Disk (DVD), memory devices, etc.) of astationary or portable program product apparatus or device for use withstand-alone systems or systems connected by a network or othercommunications medium.

The communication network may be implemented by any number of any typeof communications network (e.g., LAN, WAN, Internet, Intranet, VirtualPrivate Network (VPN), etc.). The computer or other processing systemsof the present embodiments may include any conventional or othercommunications devices to communicate over the network via anyconventional or other protocols. The computer or other processingsystems may utilize any type of connection (e.g., wired, wireless, etc.)for access to the network. Local communication media may be implementedby any suitable communication media (e.g., LAN, hardwire, wireless link,Intranet, etc.).

Each of the elements described herein may couple to and/or interact withone another through interfaces and/or through any other suitableconnection (wired or wireless) that provides a viable pathway forcommunications. Interconnections, interfaces, and variations thereofdiscussed herein may be utilized to provide connections among elementsin a system and/or may be utilized to provide communications,interactions, operations, etc. among elements that may be directly orindirectly connected in the system. Any combination of interfaces can beprovided for elements described herein in order to facilitate operationsas discussed for various embodiments described herein.

The system may employ any number of any conventional or other databases,data stores or storage structures (e.g., files, databases, datastructures, data or other repositories, etc.) to store information. Thedatabase system may be implemented by any number of any conventional orother databases, data stores or storage structures to store information.The database system may be included within or coupled to the serverand/or client systems. The database systems and/or storage structuresmay be remote from or local to the computer or other processing systems,and may store any desired data.

The embodiments presented may be in various forms, such as a system, amethod, and/or a computer program product at any possible technicaldetail level of integration. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects presented herein.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a RAM, a ROM, Erasable Programmable ROM(EPROM), Flash memory, a Static RAM (SRAM), a portable CD-ROM, a DVD, amemory stick, a floppy disk, a mechanically encoded device, and anysuitable combination of the foregoing. A computer readable storagemedium, as used herein, is not to be construed as being transitorysignals per se, such as radio waves or other freely propagatingelectromagnetic waves, electromagnetic waves propagating through awaveguide or other transmission media (e.g., light pulses passingthrough a fiber-optic cable), or electrical signals transmitted througha wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a LAN, a WAN, and/or awireless network. The network may comprise copper transmission cables,optical transmission fibers, wireless transmission, routers, firewalls,switches, gateway computers and/or edge servers. A network adapter cardor network interface in each computing/processing device receivescomputer readable program instructions from the network and forwards thecomputer readable program instructions for storage in a computerreadable storage medium within the respective computing/processingdevice.

Computer readable program instructions for carrying out operations ofthe present embodiments may be assembler instructions,Instruction-Set-Architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Python, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a LAN or a WAN, or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider). In some embodiments,electronic circuitry including, for example, programmable logiccircuitry, FPGA, or Programmable Logic Arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects presented herein.

Aspects of the present embodiments are described herein with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to the embodiments.It will be understood that each block of the flowchart illustrationsand/or block diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerreadable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, or portion of instructions,which comprises one or more executable instructions for implementing thespecified logical function(s). In some alternative implementations, thefunctions noted in the blocks may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts or carry out combinations of special purpose hardware and computerinstructions.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments. The terminologyused herein was chosen to best explain the principles of theembodiments, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

In one form, a method is provided. The method comprises: obtaining anindication of a time during which a network communication obtained froma first network node was processed by the first network node, and anindication of a propagation delay from a second network node to thefirst network node; determining a time during which the networkcommunication was processed by the second network node; calculating apropagation delay from the first network node to the second network nodebased on the time during which the network communication was processedby the first network node and the time during which the networkcommunication was processed by the second network node; determining adifference between the propagation delay from the first network node tothe second network node, and the propagation delay from the secondnetwork node to the first network node; and compensating for thedifference between the propagation delay from the first network node tothe second network node, and the propagation delay from the secondnetwork node to the first network node.

In one example, determining includes determining that the propagationdelay from the first network node to the second network node is lessthan the propagation delay from the second network node to the firstnetwork node; and compensating includes delaying the networkcommunication in an egress buffer of the second network node for anamount of time substantially equal to the difference between thepropagation delay from the first network node to the second networknode, and the propagation delay from the second network node to thefirst network node

In one example, determining includes determining that the propagationdelay from the first network node to the second network node is greaterthan the propagation delay from the second network node to the firstnetwork node; and compensating includes refraining from delaying thenetwork communication in an egress buffer of the second network node.

In one example, the method further comprises: providing, to the firstnetwork node, an indication of the propagation delay from the firstnetwork node to the second network node.

In one example, the method further comprises: providing, to the firstnetwork node, an indication of a time during which another networkcommunication was processed by the second network node to enable thefirst network node to calculate the propagation delay from the secondnetwork node to the first network node.

In one example, obtaining includes obtaining the network communication,wherein the network communication includes the indication of the timeduring which the network communication was processed by the firstnetwork node, and the indication of the propagation delay from thesecond network node to the first network node.

In one example, the method further comprises: obtaining an indication ofa time during which another network communication obtained from thefirst network node was processed by the first network node; determininga time during which the other network communication was processed by thesecond network node; calculating an updated propagation delay from thefirst network node to the second network node based on the time duringwhich the other network communication was processed by the first networknode and the time during which the other network communication wasprocessed by the second network node; determining a difference betweenthe updated propagation delay from the first network node to the secondnetwork node, and the propagation delay from the second network node tothe first network node; and compensating for the difference between theupdated propagation delay from the first network node to the secondnetwork node, and the propagation delay from the second network node tothe first network node.

In one example, the first network node includes a first local time ofday and the second network node includes a second local time of day, andthe first local time of day and the second local time of day arealigned.

In one example, the first network node and the second network node format least part of a private line emulation network.

In one example, compensating includes compensating for an asymmetry oflengths of a first optical fiber and of a second optical fiber betweenthe first network node and the second network node, a load of a networkof which the first network node and the second network node are a part,a design of the network, a temperature of the first optical fiber, or atemperature of the second optical fiber.

In another form, an apparatus is provided. The apparatus comprises: amemory configured to store executable instructions; and one or moreprocessors coupled to the memory, wherein the one or more processors areconfigured to: obtain an indication of a time during which a networkcommunication obtained from a first network node was processed by thefirst network node, and an indication of a propagation delay from asecond network node to the first network node; determine a time duringwhich the network communication was processed by the second networknode; calculate a propagation delay from the first network node to thesecond network node based on the time during which the networkcommunication was processed by the first network node and the timeduring which the network communication was processed by the secondnetwork node; determine a difference between the propagation delay fromthe first network node to the second network node, and the propagationdelay from the second network node to the first network node; andcompensate for the difference between the propagation delay from thefirst network node to the second network node, and the propagation delayfrom the second network node to the first network node.

In another form, one or more non-transitory computer readable storagemedia are provided. The non-transitory computer readable storage mediaare encoded with instructions that, when executed by a processor, causethe processor to: obtain an indication of a time during which a networkcommunication obtained from a first network node was processed by thefirst network node, and an indication of a propagation delay from asecond network node to the first network node; determine a time duringwhich the network communication was processed by the second networknode; calculate a propagation delay from the first network node to thesecond network node based on the time during which the networkcommunication was processed by the first network node and the timeduring which the network communication was processed by the secondnetwork node; determine a difference between the propagation delay fromthe first network node to the second network node, and the propagationdelay from the second network node to the first network node; andcompensate for the difference between the propagation delay from thefirst network node to the second network node, and the propagation delayfrom the second network node to the first network node.

The above description is intended by way of example only. Although thetechniques are illustrated and described herein as embodied in one ormore specific examples, it is nevertheless not intended to be limited tothe details shown, since various modifications and structural changesmay be made within the scope and range of equivalents of the claims.

What is claimed is:
 1. A method comprising: obtaining an indication of atime during which a network communication obtained from a first networknode was processed by the first network node, and an indication of apropagation delay from a second network node to the first network node;determining a time during which the network communication was processedby the second network node; calculating a propagation delay from thefirst network node to the second network node based on the time duringwhich the network communication was processed by the first network nodeand the time during which the network communication was processed by thesecond network node; determining a difference between the propagationdelay from the first network node to the second network node, and thepropagation delay from the second network node to the first networknode; and compensating for the difference between the propagation delayfrom the first network node to the second network node, and thepropagation delay from the second network node to the first networknode.
 2. The method of claim 1, wherein: determining includesdetermining that the propagation delay from the first network node tothe second network node is less than the propagation delay from thesecond network node to the first network node; and compensating includesdelaying the network communication in an egress buffer of the secondnetwork node for an amount of time substantially equal to the differencebetween the propagation delay from the first network node to the secondnetwork node, and the propagation delay from the second network node tothe first network node.
 3. The method of claim 1, wherein: determiningincludes determining that the propagation delay from the first networknode to the second network node is greater than the propagation delayfrom the second network node to the first network node; and compensatingincludes refraining from delaying the network communication in an egressbuffer of the second network node.
 4. The method of claim 1, furthercomprising: providing, to the first network node, an indication of thepropagation delay from the first network node to the second networknode.
 5. The method of claim 1, further comprising: providing, to thefirst network node, an indication of a time during which another networkcommunication was processed by the second network node to enable thefirst network node to calculate the propagation delay from the secondnetwork node to the first network node.
 6. The method of claim 1,wherein obtaining includes obtaining the network communication, whereinthe network communication includes the indication of the time duringwhich the network communication was processed by the first network node,and the indication of the propagation delay from the second network nodeto the first network node.
 7. The method of claim 1, further comprising:obtaining an indication of a time during which another networkcommunication obtained from the first network node was processed by thefirst network node; determining a time during which the other networkcommunication was processed by the second network node; calculating anupdated propagation delay from the first network node to the secondnetwork node based on the time during which the other networkcommunication was processed by the first network node and the timeduring which the other network communication was processed by the secondnetwork node; determining a difference between the updated propagationdelay from the first network node to the second network node, and thepropagation delay from the second network node to the first networknode; and compensating for the difference between the updatedpropagation delay from the first network node to the second networknode, and the propagation delay from the second network node to thefirst network node.
 8. The method of claim 1, wherein the first networknode includes a first local time of day and the second network nodeincludes a second local time of day, and wherein the first local time ofday and the second local time of day are aligned.
 9. The method of claim1, wherein the first network node and the second network node form atleast part of a private line emulation network.
 10. The method of claim1, wherein compensating includes compensating for an asymmetry oflengths of a first optical fiber and of a second optical fiber betweenthe first network node and the second network node, a load of a networkof which the first network node and the second network node are a part,a design of the network, a temperature of the first optical fiber, or atemperature of the second optical fiber.
 11. An apparatus comprising: amemory configured to store executable instructions; and one or moreprocessors coupled to the memory, wherein the one or more processors areconfigured to: obtain an indication of a time during which a networkcommunication obtained from a network node was processed by the networknode, and an indication of a propagation delay from the apparatus to thenetwork node; determine a time during which the network communicationwas processed by the apparatus; calculate a propagation delay from thenetwork node to the apparatus based on the time during which the networkcommunication was processed by the network node and the time duringwhich the network communication was processed by the apparatus;determine a difference between the propagation delay from the networknode to the apparatus, and the propagation delay from the apparatus tothe network node; and compensate for the difference between thepropagation delay from the network node to the apparatus, and thepropagation delay from the apparatus to the network node.
 12. Theapparatus of claim 11, wherein the one or more processors are furtherconfigured to: determine that the propagation delay from the networknode to the apparatus is less than the propagation delay from theapparatus to the network node; and delay the network communication in anegress buffer of the apparatus for an amount of time substantially equalto the difference between the propagation delay from the network node tothe apparatus, and the propagation delay from the apparatus to thenetwork node.
 13. The apparatus of claim 11, wherein the one or moreprocessors are further configured to: determine that the propagationdelay from the network node to the apparatus is greater than thepropagation delay from the apparatus to the network node; and refrainfrom delaying the network communication in an egress buffer of theapparatus.
 14. The apparatus of claim 11, wherein the one or moreprocessors are further configured to: provide, to the network node, anindication of the propagation delay from the network node to theapparatus.
 15. The apparatus of claim 11, wherein the one or moreprocessors are further configured to: provide, to the network node, anindication of a time during which the network communication wasprocessed by the apparatus to enable the network node to calculate thepropagation delay from the apparatus to the network node.
 16. One ormore non-transitory computer readable storage media encoded withinstructions that, when executed by a processor, cause the processor to:obtain an indication of a time during which a network communicationobtained from a first network node was processed by the first networknode, and an indication of a propagation delay from a second networknode to the first network node; determine a time during which thenetwork communication was processed by the second network node;calculate a propagation delay from the first network node to the secondnetwork node based on the time during which the network communicationwas processed by the first network node and the time during which thenetwork communication was processed by the second network node;determine a difference between the propagation delay from the firstnetwork node to the second network node, and the propagation delay fromthe second network node to the first network node; and compensate forthe difference between the propagation delay from the first network nodeto the second network node, and the propagation delay from the secondnetwork node to the first network node.
 17. The one or morenon-transitory computer readable storage media of claim 16, wherein theinstructions further cause the processor to: determine that thepropagation delay from the first network node to the second network nodeis less than the propagation delay from the second network node to thefirst network node; and delay the network communication in an egressbuffer of the second network node for an amount of time substantiallyequal to the difference between the propagation delay from the firstnetwork node to the second network node, and the propagation delay fromthe second network node to the first network node.
 18. The one or morenon-transitory computer readable storage media of claim 16, wherein theinstructions further cause the processor to: determine that thepropagation delay from the first network node to the second network nodeis greater than the propagation delay from the second network node tothe first network node; and refrain from delaying the networkcommunication in an egress buffer of the second network node.
 19. Theone or more non-transitory computer readable storage media of claim 16,wherein the instructions further cause the processor to: provide, to thefirst network node, an indication of the propagation delay from thefirst network node to the second network node.
 20. The one or morenon-transitory computer readable storage media of claim 16, wherein theinstructions further cause the processor to: provide, to the firstnetwork node, an indication of a time during which the networkcommunication was processed by the second network node to enable thefirst network node to calculate the propagation delay from the secondnetwork node to the first network node.